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ABSTRACT 


The Navy’s Strategic Planning and Analysis Directorate (OPNAV N14) uses a 
complex model to project officer status in the coming years. The Officer Strategic 
Analysis Model (OSAM) projects officer status using an initial inventory, historical 
loss rates, and dependent functions for accessions, losses, lateral transfers, and 
promotions that reflect Navy policy and U.S. law. OSAM is a tool for informing 
decision makers as they consider potential policy changes, or analyze the impact 
of policy changes already in place, by generating Navy Officer inventory 
projections for a specified time horizon. 

This research explores applications of data farming for potential 
improvement of OSAM. An analysis of OSAM inventory forecast variations over a 
large number of scenarios while changing multiple input parameters enables 
assessment of key inputs. This research explores OSAM through applying the 
principles of design of experiments, regression modeling, and nonlinear 
programming. The objectives of this portion of the work include identifying critical 
parameters, determining a suitable measure of effectiveness, assessing model 
sensitivities, evaluating performance across a spectrum of loss adjustment 
factors, and determining appropriate values of key model inputs for future use in 
forecasting Navy officer inventory. 
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EXECUTIVE SUMMARY 


Manpower and personnel costs consume a significant portion of the U.S. Navy 
budget every year, and Navy planners face the challenge of balancing manpower 
requirements and mandated end strength with budget constraints. The variability 
of human behavior further complicates the issue of forecasting strength. The 
Chief of Naval Personnel’s (NTs) Strategic Resourcing Branch is responsible for 
analyzing manpower inventory forecasts and estimating the Navy’s manpower 
expenditures to be included in the budget and Program Objectives Memorandum 
(POM) submitted to the Secretary of the Navy every two years. Forecasting 
Navy Officer inventory is a complex problem with an imperfect solution. One tool 
currently in use to tackle this problem is the Officer Strategic Analysis Model 
(OSAM), a model that follows individual officers from an initial inventory, or 
entities, through possible attribute changes during a forecast period, resulting in 
a projection of status for each officer in the coming years. OSAM is a tool for 
informing decision makers as they consider potential policy changes, or 
analyzing the impact of policy changes already in place, by generating Navy 
officer inventory projections for a specified time horizon. 

Navy officer inventory changes continuously, but decision makers find it 
useful to have accurate information about annual variations in inventory; this is 
what OSAM models. The key attributes of rank, designator, time-in-service, and 
time-in-grade describe each officer in inventory and adjust as time elapses. In 
addition to initial inventory, functions describing accession, promotion, lateral 
transfer, and loss influence the total officer inventory characterization. To model 
losses, OSAM multiplies historical loss rates by a loss adjustment factor for each 
category of officers. The model is capable of assigning a unique loss adjustment 
factor for each combination of key attributes, but recent practice is to apply the 
same loss adjustment factor universally to all designators and grades in a given 
projection year (i.e., loss adjustment factor = 1.041 in year two, 1.059 in year 
three, 1.085 in year four, 1.114 in year five). These values describe a set of 
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typical loss adjustment factors that OPNAV N14 provided to model a slowly 
improving economy. The thesis work presented here analyzes OSAM using 
designs of experiments and simulation analysis to explore the capabilities and 
limitations of the model, specifically targeting the lateral transfer and loss 
functions within the model. 

Deployment of OSAM on Naval Postgraduate School (NPS) computers 
enables exploitation of multiple processors and advanced statistical methods of 
analysis. Several different analytical methods provide insights into OSAM. Final 
analysis results in recommended loss adjustment factors that generate a better 
forecast than past practice, at a 90% confidence level. An OSAM modification 
allows the tracking of loss behavior of Surface Warfare Officers (SWOs) whose 
lateral transfer efforts are unsuccessful. Analysis of experiments reveals that 
these officers’ loss adjustment factors are the same as SWOs who never applied 
for lateral transfer, at a 95% confidence level. Additional experiments are 
necessary to determine whether there is a benefit to modeling these officers 
separately, as this OSAM modification allows a unique historical loss rate to 
enter the model. 

This thesis compares OSAM forecasts to historical inventories over a five 
year period, and results could potentially improve with additional base year data 
available. Extension of this analysis over a broader time frame may also capture 
the impact of varying political and economic environments. This first foray into 
data farming OSAM studies only two officer communities, including four 
designators. Recommendations for future research include variation of additional 
parameters in future designs of experiments, and focused analysis on individual 
community forecasts, independently or in conjunction with other communities or 
the overall officer inventory. A weighted approach to such an analysis can require 
satisfaction of absolute requirements while observing flexibility in the officer 
inventory distribution according to anticipated behavior. 
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I. INTRODUCTION 


A. PROBLEM STATEMENT 

Manpower and personnel costs consume a significant portion of the U.S. 
Navy budget every year, and Navy planners face the challenge of balancing 
manpower requirements and mandated end strength with budget constraints. 
The variability of human behavior further complicates the issue of forecasting 
strength. The Chief of Naval Personnel’s (Nl’s) Strategic Resourcing Branch is 
responsible for analyzing manpower inventory forecasts and estimating the 
Navy’s manpower expenditures to be included in the budget and Program 
Objectives Memorandum (POM) submitted to the Secretary of the Navy every 
two years. Forecasting Navy Officer inventory is a complex problem with an 
imperfect solution. One tool currently in use to tackle this problem is the Officer 
Strategic Analysis Model (OSAM), an entity based model that follows individual 
officers, or entities, from an initial inventory through possible attribute changes 
during a forecast period, resulting in a projection of status for each officer in the 
coming years. OSAM is not a tool for predicting budgetary expenses, but rather 
for informing decision makers as they consider potential policy changes, or 
analyzing the impact of policy changes already in place, by generating Navy 
officer inventory projections for a specified time horizon. 

In its current form, OSAM is complicated to use and limited to small-scale 
employment. Prior to conduct of this research, OSAM had not been analyzed for 
variations over a large number of scenarios while changing multiple input 
parameters. The application of data farming methodology to the exploration of 
OSAM enhances understanding of the model’s capabilities and limitations. This 
thesis focuses on identifying effective means of applying data farming tools to 
OSAM exploration while targeting specific research questions related to the loss 
and lateral transfer functions of the model. 
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B. PURPOSE 


This study determines whether an adaptation of the OSAM currently in 
use by the Navy’s Strategic Planning and Analysis Directorate (OPNAV N14) 
could more effectively inform decisions about future loss rates and lateral 
transfers. OSAM forecasts are particularly sensitive to changes in projected loss 
rates, and there is not an accurate means to project loss rates. A robust design 
of experiments and iterative data analysis results in a better understanding of 
how variations in specific loss rates and lateral transfer rates affect OSAM 
inventory forecasts. Application of these methods to OSAM yield a more 
beneficial planning tool for decision makers. 

There is a great deal of interest in retention strategies for specific 
communities, in addition to officer retention in the combined officer corps. Lateral 
transfers are essential to staff communities that have no direct accession 
sources, and to retain talented leaders that otherwise might leave the Navy 
altogether. This study assesses the sensitivity of OSAM to variations in loss rate 
projections and lateral transfer rates, and suggests a systematic approach for 
future researchers to apply to additional questions. 

An additional product of this research is the modification of OSAM 
scenario management for a more efficient and user-friendly interface. Automated 
model implementation for data farming opens up a wealth of future research 
opportunities. Extending these tools to examination of additional aspects of 
officer inventory management may lead to improved community management, 
accession planning, promotion planning, and loss rate projections. 

C. RESEARCH QUESTIONS 

Navy officer inventory changes continuously, while long-term planning 
relies on effective personnel forecasting to influence key decisions. OSAM fills an 
essential role for Navy planners by modeling the Navy officer component of 
personnel forecasting. In this model, the key attributes of rank, designator, time- 
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in-service, and time-in-grade describe each officer in inventory and adjust as time 
elapses. In addition to initial inventory, functions describing accession, 
promotion, lateral transfer, and loss influence the total officer inventory 
characterization. This research focuses on the outcome of lateral transfers and 
losses on Navy Officer inventory projections by addressing three specific 
research questions: 

• What is a reasonable range of loss adjustment factors to use in 
OSAM for accurate officer inventory projections? 

• How sensitive are officer inventory projections to varying future loss 
rates for specific communities or pay grades? 

• Is there a forecasting benefit to adjusting loss rates differently for 
officers who applied for lateral transfer and were declined, 
compared to officers who never applied for lateral transfer? 

In answering these research questions, the application of specific 
hypothesis tests yields insight to analytical results. These hypotheses, discussed 
at length in Chapter IV, are: 

• Primary hypothesis 1: The forecast generated from experimentally 
determined loss adjustment factors is more accurate than a 
forecast generated from a set of loss adjustment factors in which all 
values are 1.0. 

• Primary hypothesis 2: The forecast generated from experimentally 
determined loss adjustment factors is more accurate than a 
forecast generated from a set of loss adjustment factors in which 
values for projection year one are equal to 1.0, year two values are 
1.041, year three values are 1.059, year four values are 1.085, and 
year five values are 1.114. These values describe a set of typical 
loss adjustment factors that OPNAV N14 provided to model a 
slowly improving economy. 
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• Secondary hypothesis: Applying the unique set of loss adjustment 
factors for 902x officers, SWOs with declined lateral transfer 
applications, determined from a rigorous analysis yields a more 
accurate forecast than applying SWO loss adjustment factors to 
these declined lateral transfer applicants. 

D. SCOPE AND METHODOLOGY 

This study examines Navy officer inventory projection using the three 
research questions. Automation of input parameter adjustment and the post¬ 
processing of results enabled an approach to the research questions in the 
context of data farming. Analysis of preliminary experiments identifies the loss 
rates with a significant impact on OSAM forecast accuracy and narrows the 
relevant range of these loss rates, resulting in a more compact design space for 
the next set of experiments. Selection of the final design space maximizes the 
information obtained about the model within the constraints of available time and 
computing power. 

OPNAV N14 enhanced OSAM to track lateral transfer applicants not 
selected for transfer, assigning a distinct loss rate to these officers. Analysis of 
preliminary and final experiments compares experimental results to historical 
data to determine whether the change to OSAM for tracking lateral transfer 
applicants results in more accurate officer inventory projections. Additionally, 
experiments conducted for various loss rates were compared to historical data to 
determine what range of loss adjustment factors should be used by OSAM 
operators in future scenario consideration. Ultimately, this study demonstrates 
the value of OSAM and its potential analytical products in a data farming 
environment. 
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E. 


ORGANIZATION OF THESIS 


This thesis contains five chapters. Chapter II presents a brief overview of 
the current Navy Officer manpower planning process, the role of Officer Strategic 
Analysis Model (OSAM) in personnel policy decisions, and how data farming can 
multiply the power of OSAM as a tool for Navy decision makers. Chapter II also 
reviews manpower inventory projection related literature. Chapter III presents the 
design of experiments employed in data farming OSAM, and discusses the 
methodology of the data farming process. Chapter IV presents the analysis of 
experimental results. Chapter V reports conclusions and recommendations for 
future research. 
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II. BACKGROUND 


This chapter develops the foundation necessary to understand the 
importance of this thesis to United States Navy decision makers. In addition to 
defining the Navy officer manpower environment and reviewing past research 
conducted in this field, the background contained herein provides context to the 
methodology and results detailed in later chapters. 

A. NAVY OFFICER MANPOWER 

A key component of budgeting is accurately predicting manpower 
inventory in a dynamic environment subject to both organizational policies and 
the desires of individuals within the system. Officer inventory is particularly 
difficult to predict, as personnel may enter the system at multiple ranks, can 
choose to move between specialties at numerous points in a career, and there is 
no proven means to predict officer loss rates. 

U.S. Code, Title 10 regulates original appointments for commissioned 
officers (Section 33), specifies the control of officers above the grade of 0-3 
(Section 32), and governs aspects of officer promotions, separations, and 
involuntary retirements for all United States military branches (Section 36). Title 
10 codifies some facets of the Defense Officer Personnel Management Act 
(DOPMA), passed into law in 1980. In addition to aspects of DOPMA written into 
law, Department of Defense (DoD) policies derived from Congressional intent 
supplementing DOPMA legislation provide guidelines on promotion flow points 
and desirable minimum promotion opportunity for officers in each pay grade. 
(Schirmer et al., 2006) 

Officer career progression occurs upwardly via promotions, and 
sometimes laterally via redesignation. Lateral transfers between Navy officer 
communities potentially improve retention and career satisfaction for individuals, 
and enhances the ability of the Navy to staff officer communities properly when 
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unexpected shortfalls occur. According to Chief of Navy Operations (CNO) 
Instruction (OPNAVINST 1210.5), dated 24 Dec 2005, the purpose of lateral 
transfers is “to provide flexibility in the manning of officer communities.” This 
policy offers flexibility to the Navy, but also to individual officers, and adds 
additional uncertainty to the prediction of loss rates for applying officers denied 
the opportunity to transfer. 

Officer inventory projection models incorporate the key elements of officer 
career management and progression described here. Such models provide 
allowances for deviations from policies and practices to enable evaluation of 
policy changes, while intractable rules integrate laws into the modeling 
environment. A quick reference guide published for the Strategic Planning and 
Analysis Directorate (OPNAV N14) delineates which aspects of personnel 
management meet law, which satisfy policy, and which implement common 
practice (Yardley et al., 2005). 

B. OFFICER STRATEGIC ANALYS IS MODEL (OSAM) 

1. Overview of the Model 

OPNAV N14 uses OSAM to project inventory of the active duty Navy 
Officer Corps over the time horizon of the Officer Programmed Authorization 
(OPA). The OPA specifies how many officers of each grade (Ensign through 
Captain) are required in each designator (73 skill sets modeled) in each 
projection year across the Fiscal Year Defense Plan (FYDP), which extends six 
years into the future. OSAM is an entity based model, maintaining designator, 
grade, and time-in-service information for individual officers, current and 
projected. By modeling officer behavior consistent with how the Navy Officer 
Corps behaves, OSAM attempts to generate a supply of officers with the right 
skills in the right grades at the right times. 

The purpose of OSAM is to predict on a yearly basis the grade, skill, and 
time-in-service (or years of commissioned service) content of the Active Navy 
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Officer force. In the model, four inter-related functions influence expected 
outcomes: loss generation, promotions, lateral transfers, and accessions. User- 
defined parameters affect each of these functions, and they influence each other 
through interactions within the model. One objective of OSAM is to inform policy 
decisions by generating specific scenarios on request for decision makers. For 
instance, if Navy Recruiting Command needs to know whether their current 
accessions plan, in combination with NROTC and Naval Academy accessions, 
will be sufficient to supply the warfare communities with the required lieutenants 
in four years, OSAM can execute a scenario with the given accession plan to 
answer this question. A user-defined setting determines whether OSAM runs 
constrained to the inventory set by OPA for each year, or unconstrained to 
observe how officer inventory behaves on its own. The output of OSAM is a 
complete inventory of officers at the end of each projection year, characterized 
by the attributes of grade, designator, date of rank, time-in-service, and years of 
commissioned service. 

OSAM has great potential to inform policy makers, but it has many 
weaknesses as well. As a deterministic model, OSAM provides no confidence 
intervals for the officer inventory projected by its unconstrained mode. 
Furthermore, OSAM utilizes four interdependent functions to generate final 
output, and the interactions between these function may change from projection 
year to projection year. Of these four functions, promotions and accessions 
closely reflect Navy policies and behaviors. The promotion function is modeled 
three different ways, with user input determining the model to use in a given 
scenario; the default option is to promote to vacancy, as this is the underlying 
Navy policy. A known accession plan models officer gains for the first projection 
year, and in subsequent years new accessions are either unconstrained 
(determining according to the promotions, losses, and lateral transfers) or 
constrained so that officer end strength equals OPA. Promotion and accession 
functions in OSAM closely reflect the policies and practices implemented by 
Navy decision makers. Losses and lateral transfers are functions that depend 
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heavily on the unpredictable behavior and personal decisions of individual 
officers. The focus of this research is on the OSAM functions for loss rates and 
lateral transfers, intending to identify model parameters than will accurately 
capture the average behavior of individuals. 

2. Modeling Officer Loss Rates 

OSAM operators at OPNAV N14 note that the most significant unknown 
factor on model output is loss rates for projection years. Model output appears 
more sensitive to changes in loss rates than to other parameters, yet there is no 
good prediction system for officer loss rates from the Navy (as opposed to 
enlisted loss rates, which are correlated closely with unemployment rates). 
OSAM generates losses in two ways: regular loss rates reflect officers leaving 
the Navy by personal choice or policy reasons, and forced loss rates reflect the 
practice of from active service if twice non-selected for promotion. For any 
scenario, a user can also elect to turn on or off the force out business rules, 
potentially informing decisions on when and how often to allow exceptions to that 
rule. 

OSAM generates regular loss rates by multiplying historical loss rates 
(either most recent year baseline or a three-year average from the preceding 
three years) for a particular designator/rank/time-in-service combination by a loss 
adjustment factor defined by the OSAM user. For a particular 
designator/rank/time-in-service combination, the loss adjustment factor for a 
given projection year is equal to 1.0 if the analyst believes losses in that 
projection year will be exactly equal to historical losses. The adjustment factor 
will be less than 1.0 if the analyst believes the losses will be less than historical 
losses, and greater than 1.0 if the expected losses are greater than historical 
losses. Without a clear prediction formula for officer loss rates in future years, a 
working set of parameters for hypothesis testing is to apply the same loss 
adjustment factor universally to all designators and grades in a given projection 
year (i.e., loss adjustment factor = 1.041 in year two, 1.059 in year three, 1.085 in 
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year four, 1.114 in year five). These values describe a set of typical loss 
adjustment factors that OPNAV N14 provided to model a slowly improving 
economy. 

Each scenario implemented in OSAM has 2,025 loss adjustment factors 
as inputs to the model. Prior to this research, common practice applied the same 
loss adjustment factor to all officers in a given projection year, when in fact the 
loss rates of pilots five years from now may have changed differently than the 
loss rates of intelligence officers. While the loss adjustment factors used are 
reasonable values based on the experience of OSAM designers and users, it is 
desirable to know just how sensitive the model is to these changes. Data farming 
provides an opportunity to observe a variety of loss rates in a range, and to 
consider how varying loss rates differently for individual designator and grade 
combinations could affect total officer inventory and individual community 
inventory. 

With access to the actual Navy officer inventory data for every fiscal year 
since 1978, it is feasible to select multiple start years, and use OSAM to project 
FYDP inventory with various loss adjustment factors, then compare projections to 
the actual historical inventories in projected years. Rather than varying all 2,025 
loss rates, this research focuses on surface warfare officers (SWOs) and human 
resources (HR) officers. 

A third designator, 902x, created in OSAM to support this research, 

represents SWOs declined an opportunity to lateral transfer. Past research has 

shown that these officers have a significantly higher probability of leaving the 

Navy than SWOs who successfully lateral transfer or never apply for a lateral 

transfer (Kleyman & Parcell, 2010). Each officer modeled in OSAM belongs to 

only one community at a time. In reality, 902x officers as defined for this research 

are a subset of the SWO community, but in OSAM these two groups are disjoint 

sets. Even limiting variation of loss adjustment factors to these three categories, 

each design point includes 90 variables representing a unique loss adjustment 

factor in OSAM. A preliminary design of experiments determines which loss rates 
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are most useful to vary, and subsequent iterations further narrow the range of 
loss rates appropriate to use in forecasting officer inventory over multiple start 
years. 


3. Modeling Lateral Transfers 

There are three types of lateral transfers modeled in OSAM: training 
attrites, option officers, and lateral transfers. Training attrites are officers who do 
not complete initial training in their original designator. Option officers enter the 
Navy with a contract specifying a lateral transfer to occur after completing 
surface warfare officer qualification. The transfer behavior of these two groups is 
explicitly modeled in a user defined input file; an OSAM user can specify how 
officers move among designators via the lateral transfer application process, but 
the typical user setting is to leave lateral transfer behavior unconstrained, letting 
OSAM calculate the correct number. In this case, OSAM determines the number 
of officers required to supply lateral transfers to recipient communities. Each 
supplying community supplies a fraction of all its officers eligible for lateral 
transfer such that each community supplies a similar proportion of its whole. 
Notably, there is no restriction modeled in OSAM on the number of officers taken 
from each supplying community, which could potentially leave one or more of 
these supplying communities short of senior officers in a future projection year. 

A 2010 study of the lateral transfer application process by Center for 
Naval Analyses (CNA) observes that 41% of applicants disapproved for lateral 
transfer leave the Navy within 36 months, while only 10% of applicants approved 
for lateral transfer leave the Navy within 36 months. (Kleyman & Parcell) 
Currently, OSAM does not model the lateral transfer application process itself, 
only the change in designator for the officers approved to transfer. The findings 
of CNA could have a significant impact on the total Navy officer inventory and in 
particular on specific officer communities. Kleyman and Parcell’s (2010) study 
consider the applicants to lateral transfer boards in the period between 2005 and 
2010. If applied to an improved OSAM model, their findings enable comparison 
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of OSAM forecasts for multiple start years to the original OSAM model 
projections, and to the actual observed historical inventories. 

4. Goals of the Model 

“All models are wrong, but some are useful” (Box & Draper, 1987). OSAM 
is a complex model used to observe the results of “What if?” scenarios for Navy 
planners. At this time, OSAM is a Microsoft Visual FoxPro 9.0 executable 
application, dependent on input contained in 60 database files. A baseline 
scenario draws information from each of these files, and adjustments to the 
baseline scenarios occur via individually editing one or more of the 60 input files. 
A Microsoft Word document maintained with each set of database files tracks 
specific parameters defined in the database files. Generating a new scenario 
involves meticulous attention to detail in adjusting the database files and 
documenting the changes. Some methods employed to reduce human error 
when using this complex scenario management system are repetition and 
consistent practices. There is a single employee at OPNAV N14 whose primary 
job is to track and maintain OSAM scenarios, and to run new scenarios as 
needed. Even this experienced individual requires fifteen to twenty minutes to 
prepare and document a well-defined scenario. The run itself takes 3-12 
minutes, depending on the machine resources available. Output analysis, 
primarily visualization in pivot charts, occurs immediately after running a scenario 
and takes 15-30 minutes, depending on the number and types of questions of 
interest. Despite complexities in implementation, OSAM is a powerful tool with 
great potential for informing Navy leadership on the impacts of proposed policy 
changes. 

This thesis, in conjunction with the Simulation Experiments & Efficient 
Designs (SEED) Center for Data Farming, considers different choices of input 
parameters to OSAM within reasonable ranges, performs sensitivity analysis on 
these parameters, and suggests reasonable values or ranges of values to use as 
inputs to OSAM. Preliminary analysis determines reasonable ranges of variation 
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for ninety loss adjustment factors. The input to OSAM that has the greatest 
influence on results is loss rates, for which there is no reliable prediction means 
at this time. The ability to execute numerous runs of OSAM in a short period in a 
data farming environment provides an opportunity to understand the impact of 
varying loss rates on officer inventory. 

C. LITERATURE REVIEW 

For as long as there have been organizations, managers have had an 
interest in predicting the flow of manpower into, within, and out of them; military 
leaders have faced this challenge for as long as armies have been around. This 
need for leaders to plan ahead evolved over time to simple manpower modeling 
and then to multifaceted projection planning as the size and complexity of 
organizations expanded over the centuries. Manpower modeling in general and 
military manpower modeling in particular has been the subject of targeted 
operations research since before operations research has been a recognized 
discipline. As availability of computing power has improved dramatically, 
manpower modeling has reached new levels of complexity; the research 
conducted over the past few decades has laid a solid foundation for the Officer 
Strategic Analysis Model examined in this thesis. 

A review of literature previously published on the subject of manpower 
modeling provides background information on the subject, identifies successful 
model explorations, and highlights limitations of previous studies in the context of 
this thesis. Publications on the subject of manpower modeling are numerous, but 
none apply data farming tools to exploring an existing officer manpower model. 
This literature review identifies research on similar or related topics, such as 
developing new officer inventory models, or data farming an existing enlisted 
manpower model. Despite the significant gap between this thesis and previously 
published works, each of the sources identified in this literature review provide 
background and context to the evolving field of forecasting military personnel 
accurately. 
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The results of this literature review cover two distinct topics: Navy 
manpower modeling, and other service manpower modeling. The discussion on 
Navy manpower modeling falls under the subdivisions of inventory projection 
research and studies targeting lateral transfer processes and loss rate 
projections. 

1. Navy Manpower Modeling 

a. Inventory Projection 

One definition of manpower planning is the interaction of three 
processes: predicting future demand, predicting future supply, and evaluating 
policies intended to bring predicted demand and predicted supply as close 
together as possible (Edwards, 1983). Inventory projection brings together all 
three of these functions by using organizational knowledge and policies to predict 
supply and demand, and observing the gap between them. Each inventory 
projection model considered in this literature review has the ultimate goal of 
informing decision makers on the impact of potential policy changes. 

Clark (2009) develops a linear optimization program, 
Requirements-Driven Cost-Based Manpower Optimization (RCMOP), to project 
monthly values for Navy officer inventory by minimizing unmet manpower 
requirements without over-executing the budget. Like OSAM, Clark (2009) 
models officer inventory as a function of four component functions: promotions, 
accessions, lateral transfers, and losses. RCMOP assigns penalties to unmet 
manpower requirements to determine priority in executing the budget, while 
OSAM can either constrain inventory to meet manpower requirements or allow 
the officer inventory to progress naturally according to expected behavior of 
individuals if unconstrained by OPA. 

Wheeler (2010) builds on Clark’s 2009 RCMOP model. Wheeler 
considers variation of loss rates and additional officer communities, as well as 
increasing the time horizon considered. The majority of officer communities 
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remain in the “other” category under Wheeler’s analysis. The nature of OSAM as 
a simulation model allows incorporation of each Navy officer community 
separately to obtain high-resolution inventory. Clark (2009) and Wheeler (2010) 
focus on meeting budget requirements, a goal distinctly different from that of this 
research, which focuses on assessing the treatment of loss rates employed in 
generating an accurate forecast. 

b. Lateral Transfer Modeling or Analysis 

In 1997, the Center for Navy Analyses (CNA) examines the Navy’s 
lateral transfer system. Moore and Reese (1997) assess how the policy of 
staffing Restricted Line (RL) and Staff Corps communities fit into a strategy to 
increase retention and career satisfaction among officer. At the time of Moore 
and Reese’s study, there is a shortage of lateral transfers to sustain the RL and 
Staff communities; a secondary goal of their research is to determine whether an 
adjustment to transfer rates is appropriate. This CNA study observes that lateral 
transfer arising from training attrition early in an officer’s career fail to obtain 
warfare qualification more often than officers originally placed in a designator do; 
Moore and Reese suggest that this is due to a mismatch between the job and the 
officer. The observations in this 1997 analysis lay the foundation for numerous 
studies on the subject of lateral transfers in subsequent years, but do not attempt 
to model lateral transfers in a forecasting environment. 

In 2004, Monroe and Cymrot analyze a proposed policy of cutting 
Navy officer accessions and limiting lateral transfers, ultimately determining that 
reducing Surface Warfare Officer (SWO) accessions by 160 officers each year 
could save the Navy $91 million. This study considers only the SWO community, 
using productivity measures to estimate the tradeoffs between keeping officers in 
the SWO community and allowing them to transfer in greater numbers to RL or 
Staff communities. Generally, any SWO queried in an unofficial context will state 
that fellow SWOs pursuing a lateral transfer and declined the opportunity to 
transfer often elect to leave the Navy, thus still creating a gap in SWO 
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manpower. Monroe and Cymrot (2004) take steps to quantify these choices by 
SWOs, and lay the groundwork for future studies to drill down further. Monroe 
and Cymrot do not address the questions undertaken in this thesis, but 
nonetheless provide background important to understanding and interpreting 
forecast results. 

In 2007, Ryan identifies factors that lead Unrestricted Line (URL) 
officers to request lateral transfers, identifies lateral transfer selection criteria 
used by selection boards, and determines through regression analysis that 
officers turned down by a lateral transfer selection board have a different 
retention likelihood than officers who apply and are selected. “Officers who apply 
for lateral transfer but are not selected are more than twice as likely to leave the 
Navy as those who are selected” (Ryan, 2007, p. 73). Ryan’s research does not 
quantify the impact of his findings on inventory projections, as this thesis begins 
to do. 

Building on Ryan’s 2007 work, Kleyman and Parcell (2010) conduct 
a thorough statistical analysis of lateral transfer applicants over a five-year 
period, and determine a lower bound impact of denying applicants solely based 
on supplying community quotas. Ryan (2007) and Kleyman and Parcell (2010) 
note retention differences, but stop short of applying these differences to 
modeling loss rates for lateral transfer applicants, a task uniquely suited to data 
farming an inventory projection model such as OSAM. 

2. Other Service Manpower Modeling 

In 1983, a simulation model for military personnel analysis, the Accession 
Supply Costing and Requirements (ASCAR) model, develops to compute 
projected shortfalls in desired end strength and total man-years (Collins et al., 
1983). This model, like OSAM, informs decision makers on the impact of 
potential personnel policies. This model is an important predecessor to OSAM in 
both its design and its purpose, but as a model for active duty enlisted personnel, 
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ASCAR does not consider the complexities of prior service and promotions in 
controlled grades, which this thesis must incorporate in the analysis of officer 
inventory projections. 

In 1991, Grant publishes a special report discussing a non-line officer 
projection model developed for the United States Air Force to inform the effect of 
policy decisions, such as compensation and promotion adjustments, on future 
force structure. This model is an discrete event simulation and includes many of 
the functional aspects that OSAM addresses, though there is no allowance for 
lateral transfers in this Air Force model. Notably, the non-line officer projection 
model presented by Grant (1991) does not consider the non-line officer force as 
an aggregate. That is, the model provides a forecast for each officer community 
in isolation, thus missing any interaction effects that affect the total force. 

Fiebrandt (1993) develops and implements a U.S. Coast Guard Rating 
Forecast Model to project inventory and personnel flow by rating. Though this 
model is for enlisted personnel and for a different military service, it treats 
functions similarly to OSAM, though the constraints behave differently. The 
greatest shortcoming of Fiebrandt’s model in the context of officer inventory 
projection is that, like Grant (1991), it models only one community at a time, 
losing the information gained from observing redesignations between ratings. 

In 2002, Schrews develops a new optimization model for enlisted 
manpower projection in the United States Army Active Guard Reserve (AGR). 
The model developed by Schrews (2002) tracks soldiers through a simulation 
with the same attributes that OSAM follows, and deals with accessions at pay 
grades E-4 to E-9, rather than at the initial training point. Though Schrews (2002) 
does address loss rate variations, these variations have a small impact on 
results, as new gains to the system replacing losses require only a one-week 
training pipeline. OSAM must incorporate training pipelines from 6 months to 8 
years for different designators, so the effect of loss rates on forecasts warrants 
rigorous analysis. 
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Erdman (2010) uses experimental design and data analysis to study the 
U.S. Army’s Enlisted Specialty (ES) model. The ES model uses coefficients to 
set penalties and rewards on different components of its objective function; the 
user can adjust these coefficients to focus the optimization on certain goals to 
observe the impact on future decisions. Erdman’s 2010 thesis uses data farming 
to select the coefficient values that had the greatest impact on lowering the 
deviation between inventory and authorizations over the planning horizon. Like 
OSAM, the ES model projects manpower inventory in specific job types and 
ranks. While the modeling methods employed are very different between the two 
(entity-based simulation in OSAM vs. linear optimization in ES model), the 
applicability of data farming is similar for each. The use of historical data as a 
representation of the future to assess input parameter values is important to this 
thesis, as it was in assessing the ES model. 

While the studies reviewed in this chapter each provide insight to the 
intricacies of Navy officer manpower modeling, none of them addressed the 
issue of modeling all officers in a service, accounting for individual communities 
and transition between them. OSAM’s capability to handle these complexities is 
unique. With this fact in mind, the application of an effective design of 
experiments, data farming, and simulation analysis to OSAM is the subject of the 
remaining chapters in this thesis. 
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III. METHODOLOGY AND APPROACH 


This chapter introduces the tools and methods employed in gathering, 
processing, and analyzing data from 495 OSAM simulations. Each simulation 
produces about 80,000 rows of data, each of which represents a Navy officer in a 
given year. The selection of experimental designs and the reduction of generated 
data to a single measure of effectiveness presents a unique challenge. This 
chapter discusses analytical methods used is assessing the forecast accuracy of 
OSAM. 

A. DATA FARMING 

1. Definition and Application 

Data farming is a continually evolving combination of methods that 
capitalizes on high performance computing to explore a decision space more 
completely than traditional analytics typically allow. The opportunity to observe 
an entire landscape of solutions enables analysts to interpret results, assess 
model validity, identify and examine outliers, and explore otherwise 
insurmountable research questions (Horne & Meyer, 2010). 

Any model with multiple input parameters subject to variation could be a 
candidate for data farming. Efficient experimental design combined with 
automation of input parameter variations can leverage high performance 
computing to explore a large design space (Horne & Schwierz, 2008). Figure 1 
depicts the iterative nature of data farming. The scenario building loop in Figure 1 
includes building a model and defining research questions to explore with that 
model. This research uses a pre-built model, OSAM, so this scenario building 
loop refers to the specification of research questions and determination of how to 
proceed with model exploration. The scenario run space execution loop on the 
right hand side of Figure 1 describes the process of designating a design space, 
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executing multiple experiments, analyzing and interpreting results, and repeating 
the process as appropriate (Horne & Meyer, 2004). 



Data Farming Loop 

Figure 1. Iterative nature of data farming (From Horne & Meyer, 2004). 

A key component of effective data farming is efficient experimental design. 
Design of experiments (DOE) is a field that applies a systematic approach to 
observing the impact of changing variables in a simulation. The application of 
DOE within the data farming loop in Figure 1 enables observation of single 
variable effects and interactions between variables. The Appendix summarizes 
the components constructed to implement OSAM in a data farming environment. 

2. Design of Experiments 

Data farming enables study of a model’s response to multiple variations of 
many input parameters, available computing power and experimental design 
features magnify the benefits of data farming techniques. As one objective of 
this thesis is to explore the applicability of data farming to OSAM, the scope of 
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this experiment is limited to varying 90 parameters, as described in Tables 1-5, 
describing the loss adjustment factors of the Surface Warfare and Human 
Resources communities. All DOEs retain the mapping described in these tables, 
though in later experiments many input parameters remain constant at 
analytically determined optimal values. To assist in exploring the lateral transfer 
research question, analysts at OPNAV N14 adapted OSAM to add a new 
designator, 902x, to be defined as needed for specific research purposes. For 
this investigation, the 902x designator includes SWOs declined the opportunity to 
lateral transfer. The 902x designator has its own set of historical loss rates, 
distinct from SWOs who never applied for lateral transfer. 


Table 1. Each input parameter to vary in OSAM simulation mapping to its 
description of attributes. This table shows the mapping for the loss 
adjustment factors in the first projection year of each simulation. 


loss adj factor mapping 


Projection Year 

designator 

grade 

input parameter 



1 

VI 



2 

V2 


HR (1200) 

3 

V3 


4 

V4 



5 

V5 



6 

V6 



1 

Ml 


SWO (1110) 

2 

V8 

1 

& SWO 

3 

V9 

Trainee 

4 

V10 


(1160) 

5 

Vll 



6 

V12 


SWOs 

declined for 

Lateral 

Transfer 

(902x) 

1 

V13 


2 

V14 


3 

V15 


4 

V16 


5 

Mil 


6 

V18 
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Table 2. 


Mapping for the loss adjustment factors in the second projection year 
of each simulation. 


loss_adj factor mapping 


Projection Year 

designator 

grade 

input parameter 

2 

HR (1200) 

1 

V19 

2 

V20 

3 

V21 

4 

V22 

5 

V23 

6 

V24 

SWO (1110) 
& SWO 

Trainee 

(1160) 

1 

V25 

2 

V26 

3 

V27 

4 

V28 

5 

V29 

6 

V30 

SWOs 

declined for 

Lateral 

Transfer 

(902x) 

1 

V31 

2 

V32 

3 

V33 

4 

V34 

5 

V35 

6 

V36 


Table 3. Mapping for the loss adjustment factors in the third projection year. 


loss_adj factor mapping 


Projection Year 

designator 

grade 

input parameter 



1 

V37 



2 

V38 


HR (1200) 

3 

V39 


4 

V40 



5 

V41 



6 

V42 



1 

V43 


SWO (1110) 

2 

V44 

3 

&SWO 

3 

V45 

Trainee 

4 

V46 


(1160) 

5 

V47 



6 

V48 


SWOs 

declined for 

Lateral 

Transfer 

(902x) 

1 

V49 


2 

V50 


3 

V51 


4 

V52 


5 

V53 


6 

V54 
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Table 4. 


Mapping for the loss adjustment factors in the fourth projection year. 


loss_adj factor mapping 


Projection Year 

designator 

grade 

input parameter 

4 

HR (1200) 

1 

V55 

2 

V56 

3 

V57 

4 

V58 

5 

V59 

6 

V60 

SWO (1110) 

&SWO 

Trainee 

(1160) 

1 

V61 

2 

V62 

3 

V63 

4 

V64 

5 

V65 

6 

V66 

SWOs 

declined for 

Lateral 

Transfer 

(902x) 

1 

V67 

2 

V68 

3 

V69 

4 

V70 

5 

V71 

6 

V72 


Table 5. Mapping for the loss adjustment factors in the fifth projection year. 


loss_adj factor mapping 


Projection Year 

designator 

grade 

input parameter 



1 

V73 



2 

V74 


HR (1200) 

3 

V75 


4 

V76 



5 

V77 



6 

V78 



1 

V79 


SWO (1110) 

2 

V80 

5 

&SWO 

3 

V81 

Trainee 

4 

V82 


(1160) 

5 

V83 



6 

V84 


SWOs 

declined for 

Lateral 

Transfer 

(902x) 

1 

V85 


2 

V86 


3 

V87 


4 

V88 


5 

V89 


6 

V90 
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In a simulation environment with a small number of variable input 
parameters and a discrete number of levels for each parameter, it is possible to 
run an experiment covering every combination of variables. This set of conditions 
describes a full factorial design, and this type of experiment exhausts a design 
space, providing maximum information to an analyst. Unfortunately, it is 
infeasible to conduct a full factorial experiment on most real world models, and 
impossible for many, due to the exponential growth of design points required with 
the addition of each factor or factor level. Methods in the area called DOE devote 
resources to identifying and exploring experimental designs that cover much of 
the design space with a fraction of the design points that would appear in a full 
factorial design. Latin hypercubes (LHs) and nearly orthogonal Latin hypercubes 
(NOLHs) are two design types that satisfy this challenge neatly (Sanchez, 2006). 

When the factors, or variables, of a DOE are columns with N rows 
representing the possible levels for each factor, a randomized permutation of 
each column results in a Latin hypercube design. This type of design is most 
beneficial in experiments with large numbers of factors, and a randomized LH 
has good orthogonality properties when the number of design points is much 
greater than the number of factors. For smaller designs, it is important to select a 
LH design with low pairwise correlation between columns. An alternative to 
generating multiple LHs and choosing one with low pairwise correlations is to use 
an NOLH design. The SEED Center at NPS maintains spreadsheets pre-built to 
generate NOLH designs for small and moderate numbers of factors. Even with a 
small number of factors to vary, a full factorial design is cumbersome with more 
than two levels. An NOLH design has good space-filling and orthogonality 
properties, does not need checking for pairwise correlations, and requires a 
fraction of the design points a full factorial design would need (Sanchez, 20006). 

Running a large number of simulations, one for each design point, 
generates massive amounts of data. A relationship exists between the 
explanatory variables, or input parameters, and one or more response variables; 
determining this relationship correctly is often the goal of simulation analysis. In 
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1951, Box and Wilson introduce response surface methodology as a means to 
approximate this relationship using a second-degree polynomial. This 
methodology has proven effective and easy to apply to simulation analysis, even 
when details about the simulation process used to generate the data are 
unknown. Using current computing and available analytical software, a 
regression model allows all input parameters to enter the model as independent 
variables, along with the quadratic terms for each input parameter, and all 
possible two-way interactions between the input parameters. For an experiment 
with 90 input parameters, this methodology generates a regression model with 
an unwieldy number of terms, and additional statistical tools serve to simplify the 
model. A simulation analysis conducted after each DOE results in reduction of 
input parameters to include in the subsequent DOE if some parameters do not 
appear in the response surface model, or optimization of all generated models 
yields the same solution for some parameters. 

In conducting this research, the first DOE varies 108 factors over 108 
design points, and is a randomized LH design (S-plus script for Latin hypercubes, 
2011). Each factor is a continuous variable, but the design generation uses 108 
discrete levels for these factors. This first DOE intends to analyze variation over 
a six year forecast period, but base year data files for OSAM were only available 
as far back as 2007, so the simulation analysis was limited to a five year 
projection period. The variable mapping described in Tables 2-5 shows only the 
90 factors contributing to the analysis. Of these 90 remaining factors, five 
represent 902x officers of grade 01 (Ensigns). Navy policy prohibits officers with 
less than two years of commissioned service from applying for lateral transfer; 
this policy effectively means that 902x Ensigns never appear in the model. The 
inclusion of these five factors in the first DOE is an oversight, and subsequent 
DOEs do not consider them. 

The second DOE varies 65 factors over 128 design points, and is an 
NOLH design (Vieira, Sanchez, Kienitz, & Belderrain, 2011). The third DOE 
varies 29 factors over 128 design points, and is an NOLH design (Sanchez, 
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2005). This third DOE runs first with all 902x parameters equal to their 
corresponding SWO parameters, and a second time with all 902x parameters 
equal to values calculated in earlier analyses. Conventionally, as the number of 
factors in a DOE decreases, so does the number of design points needed. This 
research used fewer design points in earlier DOEs to explore and analyze the 
model within a reasonable period. All DOEs execute five times, once for each of 
the five base years, 2007-2011, that generate a forecast comparable to available 
historical data. 


B. DATA HANDLING 


Every design point generates about 80,000 rows of data, each 
representing a single officer in one projection year of the forecast period. To 
conduct a simulation analysis, it is necessary to condense this data into a single 
measure of effectiveness for each design point. The primary research focus is to 
assess the loss adjustments factors’ impact on forecast accuracy, so the analysis 
compares forecast inventory to actual inventory for each year that historical data 
is available. Relative difference, absolute difference, or mean squared error can 
all be effective measures of forecast accuracy, but each of these needs to be 
weighted proportionately when condensing so much data into a single value. A 
straightforward solution is to calculate the mean absolute proportional error 
(MAPE) for each design point. 


I 


MAPE , = 


actual slate - forecast st 

actual s,g,e 

N 


where N is the number of data points included. 


In the context of this thesis, the state denoted by subscript refers to the 
combination of skill (SK), grade (GR), and fiscal year (FY). To calculate MAPE 
for each design point requires significant data manipulation. First, culling the data 
to extract only SWOs, HRs, and 902x officers enables matching forecast 
inventory with actual historical data for only the designators of interest to this 
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thesis. Then, the SWO community absorbs all 902x officers for comparison with 
historical inventory. The initial redesignation of these officers to 902x allows them 
to behave uniquely in OSAM, but they are still SWOs, and must be included in 
the total SWO inventory in evaluating forecast accuracy. The remaining data 
includes a SWO and HR inventory for each projection year and grade for all 90 
design points. This table joins with a historical inventory table to generate 
MAPEfy values for each DP. The FY subscript associated with each MAPE 
value indicates that this value measures forecast accuracy for a specific fiscal 
year; each projection year OSAM generates yields a separate MAPEfy model. As 
the manipulation and combination of data tables proceeds, the number of rows, 
N, describing the data changes several times, rendering it a misleading value to 
include in final MAPE calculation. Instead, the traditional MAPE formula is 
adapted to normalize for total forecast. 


MAPE fy 


i 

SK,GR 


actual gr fy forecast SK ^ py 


actual 


SK,GR,FY 


• forecast SKGRFY 


X forecast SK ,GR,FY 

SK,GR 


The resulting data set includes one MAPEfy value for each design point, 
used in the next phase of analysis to generate regression models. Conducting 
MAPE calculations proportioned on rank and skill in this fashion ensures that a 
small group of officers, such as HR Captains (06s), do not exert the same 
influence on the overall MAPE model as a large subset, such as SWO Ensigns 
(01s). This method simultaneously ensures that the influence of deviations within 
a subset exert a proportional influence on the model (i.e., a forecast with five 
more HR Captains than historical has a greater impact on the HR Captain 
component than a forecast with five more SWO Ensigns than historical). This 
second attribute is particularly important if applying this method to modeling 
MAPE by designator and fiscal year (MAPEfy.sk), rather than only by fiscal year 
(MAPEfy) as implemented in this simulation analysis. 
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The last step in the data-handling phase of analysis is subsetting MAPE 
calculations by fiscal year, generating a file for each projection year with a MAPE 
value for each design point. Once matched with the original DOE file, this data 
table is ready for the modeling phase of analysis. 

C. REGRESSION ANALYSIS 

OSAM’s forecast accuracy is a function of many complex inter-related 
elements, none of which exhibit linear relationship in the execution of the model. 
Design of experiments and simulation analysis enable reduction of the complex 
relationship between loss adjustment factors and forecast accuracy to regression 
modeling. Ordinary least squares regression is insufficient to describe this 
relationship, but a stepwise regression allowing all terms in a response surface 
model can predict MAPE effectively. A response surface includes each 
parameter varied in the model, quadratic terms for each variable and all two-term 
interactions between variables (Box & Wilson, 1951). 

Using a minimum Akaike’s Information Criterion (AIC) stopping rule, 
stepwise regression identifies the best model, based on subjective statistical 
rules, of MAPE for each projection year and base year; each DOE generates 15 
MAPE models. The model naming convention used in this thesis is base year- 
MAPE-projection year, so the model named 2007MAPE1 is the MAPE model for 
the first projection year (FY07) of the scenario with base year 2007. 

For the first DOE, the MAPE models include a large number of terms and 
exhibit excellent fit, many with r-square values greater than 0.98. A smaller 
number of terms is desirable, so this researcher employs a process that runs the 
model iteratively to observe t-statistics for individual model coefficients and 
remove additional terms manually from the stepwise model. The modeling 
challenge is to achieve a good enough fit while limiting the total number of terms 
included in the model. The solution to this challenge is to make judgment calls in 
balancing the AIC value and goodness of fit statistics with the likelihood that 
included terms are individually significant to the model. This balancing act is 
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simpler with later DOEs, when fewer terms in the model limit the goodness of fit, 
disallowing the opportunity to remove terms from the initial stepwise model. 

D. OPTIMIZATION BY NONLINEAR PROGRAMMING 

It is desirable to minimize the absolute deviation of forecast inventory from 
actual inventory, but a direct attempt to minimize MAPE, a function of loss 
adjustment factors, will result in most or all factors being set to their minimum 
allowable values. Further, it is desirable to compare multiple MAPE models 
simultaneously. When comparing multiple models in a group, or meta-model, the 
maximum MAPE among these models is the value to minimize. Approaching the 
problem in this manner results in most or all models in the meta-model ending 
with very similar MAPE values, resulting in a better solution set than could be 
reached by minimizing the total or average MAPE of the meta-model. This 
problem formulation is: 

min mdx( MAPE ) 

xeV L xeV 

s-t- ^ mn ^ x < x max V x e {V’1,V’2,...V’90} 

MAPE > 0 V MAPE e meta-model 

The simplest meta-model to use includes each of the fifteen MAPE 
models generated for the DOE. Some base year scenarios provide MAPE data 
for only one or two years, and only one or two models cover some projection 
years. To account for this uneven distribution of data and reduce the influence of 
any single model on the final solution set, a formulation of meta-models derives 
from three different grouping categories. One category includes all models 
generated from a given base year scenario. A second category includes all 
models that project a given projection year, independent of base year. A third 
category includes all models that predict a given fiscal year. Table 6 
demonstrates the meta-model groupings considered in determining optimal 
values for each loss adjustment factor. 
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A benefit of this problem formulation is its ease of implementation in 
Microsoft Excel. A known limitation of Excel’s solver feature is that the solution is 
a local minimum, but not necessarily a global minimum. The solution set returned 
by Excel is dependent on the initial values selected for all variables. For this 
reason, the initial values chosen were consistent and equal to 1.0 for all loss 
adjustment factors. This practice does not guarantee the smallest possible MAPE 
value, but the potential errors due to this software property are smoothed by the 
averaging of all results over the fifteen meta-models described in Table 6. 
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Table 6. This table demonstrates the meta-model groupings considered in 
determining optimal values for each loss adjustment factor. The 
model naming convention used is base year-MAPE-projection year, 
so the model 2007MAPE1 is the MAPE model for the first projection 
year (FY07) of the scenario with base year 2007. The MAPE values 
in this table are for demonstration purposes only. 


grouped according to base year 

grouped according to projection 
year being forecast 
(independent of base year) 

grouped according to 
fiscal year being forecast 

2007 scenario summary 

MAPE 1 summary 

MAPE2007 summary 

model 

MAPE 

model 

MAPE 

model 

MAPE 

2007MAPE 1 

0.12369 

2007MAPE 1 

0.12369 

2007MAPE 1 

0.12369 

2007MAPE2 

0.16065 

2008MAPE 1 

0.08931 

max 

0.12369 

2007MAPE3 

0.02441 

2009MAPE 1 

0.12875 

MAPE2008 summary 

2007MAPE 4 

0.00708 

2010MAPE 1 

0.10380 

model 


2007MAPE 5 

0.11844 

2011 MAPE 1 

0.16239 

2007MAPE2 

0.16065 

max 

0.16065 

max 

0.16239 

2008MAPE 1 

0.08931 

2008 scenario summary 

MAPE 2 summary 

max 

0.16065 

model 


model 


MAPE 2009 summary 

2008MAPE 1 

0.08931 

2007MAPE2 

0.16065 

model 


2008MAPE 2 

0.05747 

2008MAPE2 

0.05747 

2007MAPE3 

0.02441 

2008MAPE3 

0.06707 

2009MAPE2 

0.09360 

2008MAPE2 

0.05747 

2008MAPE 4 

0.18762 

2010MAPE 2 

0.06817 

2009MAPE 1 

0.12875 

max 

0.18762 

max 

0.16065 

max 

0.12875 

2009 scenario summary 

MAPE 3 summary 

MAPE 2010 summary 

model 


model 


model 


2009MAPE 1 

0.12875 

2007MAPE3 

0.02441 

2007MAPE4 

0.00708 

2009MAPE2 

0.09360 

2008MAPE3 

0.06707 

2008MAPE3 

0.06707 

2009MAPE 3 

0.13666 

2009MAPE3 

0.13666 

2009MAPE2 

0.09360 

max 

0.13666 

max 

0.13666 

2010MAPE 1 

0.10380 

2010 scenario summary 

MAPE 4 summary 

max 

0.10380 

model 


model 


MAPE 2011 summary 

2010MAPE 1 

0.10380 

2007MAPE4 

0.00708 

model 


2010MAPE 2 

0.06817 

2008MAPE4 

0.18762 

2007MAPE5 

0.11844 

max 

0.10380 

max 

0.18762 

2008MAPE4 

0.18762 

2011 scenario summary 

MAPE 5 summary 

2009MAPE3 

0.13666 

model 


model 


2010MAPE 2 

0.06817 

2011 MAPE 1 

0.16239 

2007MAPE5 

0.11844 

2011 MAPE 1 

0.16239 

max 

0.16239 

max 

0.11844 

max 

0.16239 
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A solution for each of the meta-models in Table 6 determines the set of 
loss adjustment factors that minimizes the maximum MAPE in the meta-model. 
An overall meta-model including all 15 MAPE models also produces a solution 
set. These 16 solution sets demonstrate a range of values for each loss 
adjustment parameter. As discussed earlier, some parameters do not appear in 
every meta-model, but this process guarantees at least four data points for every 
loss adjustment factor. Only one MAPE model, 2007MAPE5, depends on the 
values of parameters V73 - V90. The optimization of the three meta-models in 
Table 6 including 2007MAPE5, as well as the overall meta-model, provide a 
solution set for these 18 factors, one data point from each solution set. Similarly, 
parameters V55 - V72 appear only in models 2007MAPE4, 2007MAPE5, and 
2008MAPE4, so these variables appear in seven solution sets, and their final 
recommended values depend on the seven data points available for each 
parameter in this range. These data points determine an average value and 95% 
confidence interval (Cl) for each loss adjustment factor. The Cl width for every 
factor informs the decision to hold it constant in subsequent experiments or retain 
it as an input parameter. 

The processes described in this chapter apply to analysis of data collected 
from each DOE, and the results from the first two DOEs used to formulate the 
subsequent DOE. This iterative data farming loop produces results that answer 
the research questions put forth in this thesis. Chapter IV presents a summary of 
these results. 
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IV. ANALYSIS OF EXPERIMENTAL RESULTS 


This chapter presents the computational results derived from each design 
of experiments and discusses the process of deriving subsequent experimental 
designs. This chapter also presents results of the final experiments, discusses 
the implications of these results, and assesses the statistical and practical 
significance of findings. 

A. FIRST DESIGN OF EXPERIMENTS 

1. Loss Adjustment Factors to Hold Constant 

A solution for each of the meta-models in Table 6 determines the set of 
loss adjustment factors that minimizes the maximum MAPE in the meta-model. 
An overall meta-model including all 15 MAPE models also produces a solution 
set. These 16 solution sets demonstrate a range of values for each loss 
adjustment parameter. As discussed earlier, some parameters do not appear in 
every meta-model, but this process guarantees at least four data points for every 
loss adjustment factor. 

This analysis calculates an average value and 95% confidence interval 
(Cl) for each loss adjustment factor. The factors for projection year five have the 
fewest number of data points, and thus have the widest confidence intervals, as 
a rule. A summary table of average values, the Cl lower bound (LB), and Cl 
upper bound (UB) for each factor indicates which parameters are constant in the 
next DOE. A rule of thumb used is to hold constant factors with a Cl smaller than 
0.1, or special cases of factors that do not vary much, despite having a large Cl. 
Table 7 includes factors determined by DOE1 to be constant in DOE2. The 
factors and confidence intervals in Table 7 are final determined values, and 
appear again in the graphical summary of results later in this chapter. 
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Table 7. Factors to hold constant in DOE2, based on DOE1 analysis. Shaded 
cells have a Cl wider than 0.1. V38 is held constant for DOE2 
because 9 of 11 values for V38 were equal, with outliers heavily 
influencing Cl calculation. 


Projection 

Year 

designator 

grade 

input 

parameter 

average 

95% Confidence interval 

LB 

UB 

2 

HR 

6 

V24 

1.039 

1.024 

1.054 

902x 

2 

V32 

1.000 

0.999 

1.001 

3 

HR 

2 

V38 

1.208 

1.058 

1.359 

SWO 

5 

V47 

1.000 

0.999 

1.001 

902x 

3 

V51 

1.000 

1.000 

1.000 

4 

HR 

4 

V58 

1.000 

1.000 

1.000 

5 

V59 

1.000 

1.000 

1.000 

6 

V60 

1.000 

1.000 

1.000 

SWO 

1 

V61 

1.000 

1.000 

1.000 

6 

V66 

1.000 

1.000 

1.000 

902x 

2 

V68 

1.000 

1.000 

1.000 

3 

V69 

1.000 

1.000 

1.000 

5 

HR 

1 

V73 

1.000 

1.000 

1.000 

2 

V74 

1.000 

1.000 

1.000 

5 

V77 

1.000 

1.000 

1.000 

6 

V78 

1.000 

1.000 

1.000 

SWO 

2 

V80 

1.000 

1.000 

1.000 

902x 

2 

V86 

1.000 

1.000 

1.000 

3 

V87 

1.000 

1.000 

1.000 

4 

V88 

1.000 

1.000 

1.000 


2. Selection of Parameter Ranges for DOE2 

The factors not included in Table 7 remain variable input parameters in the 
second design of experiments (DOE2). After analysis of DOE1, some factors 
have a 95% Cl greater than the size of the original range allowed in the 
experiment. The two-fold purpose of DOE1 is to identify variables to hold 
constant in subsequent experiments, and to reduce the range of remaining input 
parameters to vary in DOE2. With this in mind, the range allowed for each 
remaining variable is a maximum of 0.2. The center of each variable’s range is its 
average value determined from DOE1 analysis, with a range of 0.2 or its 95% Cl, 
whichever is smaller. 


36 




B. SECOND DESIGN OF EXPERIMENTS 


1. Loss Adjustment Factors to Hold Constant 

As in analysis of DOE1, the results of DOE2 yield a solution for each of 
the meta-models in Table 6, and an overall meta-model including all 15 MAPE 
models. Taken together, these solution sets determine the set of loss adjustment 
factors that minimizes the maximum MAPE in the meta-model. Descriptive 
statistics for these 16 solution sets determine an average value and a 95% Cl for 
each loss adjustment factor. A summary table of values and CIs for each factor 
indicates which factors are constant in the next DOE. A rule of thumb used is to 
hold constant factors with a Cl smaller than 0.1. Table 8 displays all loss 
adjustment factors to be held constant in DOE3, in addition to those indentified in 
Table 7. The shaded rows in Table 8 indicate parameters held constant in DOE3, 
despite a Cl wider than 0.1. Each of these parameters is a 902x loss adjustment 
factor, held constant in DOE3 to test the secondary hypothesis of this thesis: 
using distinct values for 902x loss adjustment factors determined from a rigorous 
analysis yields a more accurate forecast than applying SWO loss adjustment 
factors to 902x officers. The factors and confidence intervals in Table 8 are final 
determined values, and appear again in the graphical summary of results later in 
this chapter. 

2. Selection of Parameter Ranges for DOE3 

The factors not included in Table 7 or Table 8 remain variable input 
parameters in the third design of experiments (DOE3). The analysis of DOE2 
identifies variables to hold constant in subsequent experiments, and to reduce 
the range of remaining input parameters to vary in DOE3. With this in mind, all 
SWO and HR loss adjustment factors with a Cl wider than 0.1 remain variable in 
DOE3. All 902x loss adjustment factors are constant in DOE3, to test the 
secondary hypothesis of this thesis: using distinct values for 902x loss 
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adjustment factors determined from a rigorous analysis yields a more accurate 
forecast than applying SWO loss adjustment factors to 902x officers. 


Table 8. Factors to hold constant in DOE3, based on analysis of DOE2. 

Shaded cells have Cl wider than 0.1, but remain constant in the next 
DOE at their average values, for the sole purpose of testing the 
secondary hypothesis: that experimentally determined factors for 
902x will yield a better forecast than using the SWO factors for 902x. 


Projection 

designator 

grade 

input 

average 

95% Confidence interval 

Year 

parameter 

LB 

UB 


HR 

1 

VI 

1.001 

0.985 

1.018 


2 

V2 

1.004 

0.972 

1.035 



2 

V14 

0.970 

0.812 

1.127 

1 


3 

V15 

1.051 

0.882 

1.220 


902x 

4 

V16 

1.012 

0.945 

1.079 



5 

V17 

1.000 

0.964 

1.035 



6 

V18 

0.996 

0.937 

1.055 


HR 

2 

V20 

1.067 

1.051 

1.083 


3 

V21 

1.087 

1.074 

1.099 


SWO 

3 

V27 

0.883 

0.857 

0.909 

2 


3 

V33 

1.015 

0.980 

1.049 


902x 

4 

V34 

1.019 

0.963 

1.075 


5 

V35 

1.036 

0.991 

1.082 



6 

V36 

1.016 

1.013 

1.019 



1 

V37 

1.078 

1.078 

1.078 


HR 

4 

V40 

1.046 

1.030 

1.061 


5 

V41 

1.063 

1.059 

1.067 



6 

V42 

1.077 

1.067 

1.086 



1 

V43 

1.045 

1.016 

1.075 

3 

SWO 

4 

V46 

1.047 

1.011 

1.084 



6 

V48 

1.072 

1.072 

1.072 



2 

V50 

1.078 

1.051 

1.105 


902x 

4 

V52 

1.079 

1.067 

1.090 


5 

V53 

1.016 

0.982 

1.050 



6 

V54 

1.014 

0.979 

1.049 



1 

V55 

1.075 

1.075 

1.075 


HR 

2 

V56 

1.118 

1.100 

1.135 



3 

V57 

1.149 

1.149 

1.149 

4 

SWO 

4 

V64 

1.101 

1.098 

1.104 



4 

V70 

1.110 

1.110 

1.110 


902x 

5 

V71 

1.097 

1.075 

1.120 



6 

V72 

1.141 

1.056 

1.225 


HR 

4 

V76 

1.116 

1.116 

1.116 


SWO 

1 

V79 

1.183 

1.183 

1.183 

5 

6 

V84 

1.209 

1.209 

1.209 


902x 

5 

V89 

1.094 

1.094 

1.094 


6 

V90 

1.119 

1.119 

1.119 
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The remaining 28 SWO and HR loss adjustment factors vary in DOE3 
according to an NOLH design. The 128 runs of this design execute twice; DOE3a 
assigns to all 902x loss adjustment factors the value determined in analysis of 
DOE1 and DOE2, while DOE3b sets each 902x loss adjustment factor equal to 
the SWO loss adjustment factor for the same projection year and grade 
combination. 

C. THIRD DESIGN OF EXPERIMENTS 

1. Use of Unique Loss Adjustment Factors for Declined Lateral 
Transfer Applicants 

The null hypothesis for this comparison is the secondary hypothesis 
specified in Chapter I of this thesis: that using distinct values for 902x loss 
adjustment factors determined from a rigorous analysis yields a more accurate 
forecast than applying SWO loss adjustment factors to 902x officers. A paired 
two sample t-test is sufficient to test this hypothesis. The results of this t-test, 
displayed in Table 9, indicate that using experimentally determined 902x loss 
adjustment factors does not provide a more accurate forecast, as measured by 
MAPE, than setting all 902x factors equal to their corresponding SWO factors, at 
a 95% confidence level. 


Table 9. t-Test: Paired Two Sample for Means, to test secondary hypothesis: 

using distinct values for 902x loss adjustment factors determined 
from a rigorous analysis yields a more accurate forecast (smaller 
MAPE) than applying SWO loss adjustment factors to 902x officers. 



DOE3a 

DOE3b 

Mean 

0.1038115 

0.09837249 

Variance 

0.00071632 

0.00057733 

Observations 

16 

16 

Pearson Correlation 

0.93988152 


Hypothesized Mean 

0 


df 

15 


t Stat 

2.50570874 


P(T<=t) one-tail 

0.01134058 


t Critical one-tail 

1.73960672 
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This important result indicates there is no benefit to implementing a 
distinct set of loss adjustment factors for SWOs declined a lateral transfer. 
Notably, OSAM models historical loss rates differently for 902x officers than for 
SWOs who have not applied for a lateral transfer. Thus, disproval of this 
secondary research hypothesis does not mean there is no benefit to modeling 
902x officers distinctly from SWOs. Additional model exploration may offer a 
conclusive answer to this question. 

2. Final Values for Loss Adjustment Factors 

As in earlier phases of analysis, a solution for each of the meta-models in 
Table 6, and an overall meta-model including all 15 MAPE models, determines 
the set of loss adjustment factors that minimizes the maximum MAPE in the 
meta-model. Subsequent to disproval of the secondary research hypothesis, 
analysis DOE3b recommends appropriate factor values for future forecasting, 
disregarding results from DOE3a. Descriptive statistics for the 16 solution sets of 
DOE3b determine an average value and a 95% Cl for each loss adjustment 
factor. A summary table of values and CIs for each factor indicates final 
recommended values of loss adjustment factors, and their respective confidence 
intervals (See Table 10). The shaded rows in Table 10 indicate parameters with 
a Cl wider than 0.15. Additional experiments might narrow these ranges further, 
but the reduction in parameters varied through this iterative analytical process led 
to difficulty generating acceptable regression models. This research instead 
recommends accepting these average values, with a note of caution. The factors 
and confidence intervals in Table 8 are final determined values, and appear 
again in the graphical summary of results later in this chapter. 

Taken together, the values in Tables 7-8 and Table 10 are the 
experimental results determined for use in future forecasting efforts with OSAM. 
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Table 10. Factor values determined by analysis of third and final DOE. Shaded 
cells have Cl wider than 0.15. 


Projection 

designator 

grade 

input 

aMsrage 

95% Confidence interval 

Year 

parameter 

LB 

UB 



3 

V3 

0.981 

0.938 

1.024 


HR 

4 

V4 

0.984 

0.919 

1.050 


5 

V5 

1.031 

0.961 

1.101 



6 

V6 

1.020 

0.976 

1.064 

1 


1 

V7 

1.028 

0.908 

1.147 


2 

V8 

0.997 

0.897 

1.096 


SWO 

3 

V9 

0.974 

0.905 

1.043 


4 

V10 

0.999 

0.903 

1.096 



5 

Vll 

0.987 

0.903 

1.072 



6 

V12 

0.961 

0.828 

1.095 



1 

V19 

1.020 

1.018 

1.022 


HR 

4 

V22 

0.999 

0.960 

1.038 



5 

V23 

0.983 

0.957 

1.010 

2 


1 

V25 

0.991 

0.955 

1.028 


2 

V26 

1.000 

0.942 

1.057 


SWO 

4 

V28 

1.008 

0.972 

1.043 



5 

V29 

1.004 

0.931 

1.077 



6 

V30 

1.014 

0.966 

1.061 


HR 

3 

V39 

1.092 

1.005 

1.179 

3 

SWO 

2 

V44 

1.042 

1.042 

1.042 


3 

V45 

0.874 

0.794 

0.954 



2 

V62 

0.961 

0.912 

1.009 

4 

SWO 

3 

V63 

0.936 

0.831 

1.042 



5 

V65 

0.992 

0.963 

1.021 


HR 

3 

V75 

1.101 

1.082 

1.119 

5 


3 

V81 

0.902 

0.834 

0.969 

SWO 

4 

V82 

1.017 

0.981 

1.054 



5 

V83 

1.173 

1.173 

1.173 


3. Interpretation of Results 

The final experimental values displayed in Tables 7-8 and Table 10 are 
useful for implementation in OSAM, but a list of numbers is only one 
demonstration of how the recommendations of this research compare to the 
values used in OSAM runs prior to this analysis. Figure 2 is a visual 
representation of the 95% CIs for SWO loss adjustment factors, overlaid with the 
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null hypothesis (primary hypothesis 1) and the working hypothesis (primary 
hypothesis 2). The vertically aligned diamonds on this plot each represent the 
upper and lower bound of the Cl for the case indicates on the horizontal axis. 
Figure 2 shows that the Cl for many SWO loss adjustment factors contains either 
one or both hypothetical values, with some notable exceptions. 

The 03, or Lieutenant, average loss adjustment factors recommended by 
this analysis fall below the null and working hypothesis for all projection years, 
with the working hypothesis values falling completely outside the Cl for projection 
years two through five, and the null hypothesis values falling completely outside 
the Cl for projection years two, three, and five. This result could be due to a 
tendency of OSAM to overestimate 03 loss rates, leading to a recommendation 
for 03 loss adjustment factors less than 1.0 across all projection years. This 
finding may suggest a need to revisit the method of estimating 03 loss rates in 
OSAM, but more likely identifies a specific issue with SWOs, because the same 
behavior does not occur in the HR community (See Figure 3). 


95% Confidence Interval of optimal loss adjustment factors for 
SWOs compared to null and working hypotheses 
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Figure 2. Ninety-five percent confidence intervals of optimal loss adjustment 
factors for SWOs compared to null and working hypotheses. 
Diamonds indicate upper and lower bound of respective CIs. 
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Figure 3 is a visual representation of the 95% CIs for HR loss adjustment 
factors, overlaid with the null hypothesis (primary hypothesis 1) and the working 
hypothesis (primary hypothesis 2). The vertically aligned diamonds on this plot 
each represent the upper and lower bound of the Cl for the case indicates on the 
horizontal axis. Figure 3 shows that the Cl for many HR loss adjustment factors 
contains either the null or working hypothesis value. The HR factor values align 
closely with the working hypothesis values, with the exception of several values 
in projection years four and five eliminated from further variation early in analysis 
because they did not appear in any regression models. 

Figure 3 suggests that OSAM may be overestimating loss rates for HR 
02s and 03s in projection years two and four, and underestimating loss rates for 
HR 01s, 04s, and 05s in projection year two. While the deviations from 
hypothetical values are not as large as for SWOs, there is evidence to suggest 
forecast accuracy improves when using the experimentally determined loss 
adjustment factors, particularly in years two and four. 



Figure 3. Ninety-five percent confidence intervals of optimal loss adjustment 

factors for HRs compared to null and working hypotheses. Diamonds 
indicate upper and lower bound of respective CIs. 
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The number of HR officers is significantly smaller than the number of 
SWOs at any given time. This analysis determines the loss adjustments factors 
displayed in Figures 2 and 3 simultaneously. The focus of this thesis is overall 
MAPE for the conglomerate of officer communities considered, but determination 
of loss adjustment factors one designator at a time could result in detailed 
information for specific communities. Development of regression models for 
MAPE separated by designator will yield this information. 

D. VERIFICATION EXPERIMENT 

A final experiment runs with only three design points. The first DP sets all 
loss adjustment factors of interest equal to 1.0, to test primary hypothesis 1, 
referred to in Figures 2 and 3 as the null hypothesis. The second DP sets loss 
adjustment factors for projection year one equal to 1.0, 1.041 in year two, 1.059 
in year three, 1.085 in year four, and 1.114 in year five, to test primary hypothesis 
2, referred to in Figures 2 and 3 as the working hypothesis. The third DP sets all 
loss adjustment factors equal to the average values reported in Tables 7-8 and 
Table 10. Each of these DPs provides 15 MAPE values, one for each base 
year/projection year combination. These three sets of MAPE values, shown in 
Table 11, are comparable to one another for hypothesis testing. 
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Table 11. 


Summary of MAPE values from final set of OSAM runs. 



MAPE 

baseyear 

projection 

year 

hypothesis 

1 

hypothesis 

2 

Experimental 

Results 

2007 

2007 

0.03994 

0.03994 

0.04063 

2007 

2008 

0.05722 

0.05793 

0.0649 

2007 

2009 

0.08593 

0.0876 

0.09325 

2007 

2010 

0.1308 

0.13008 

0.11542 

2007 

2011 

0.16801 

0.17947 

0.15208 

2008 

2008 

0.05485 

0.05485 

0.05305 

2008 

2009 

0.07957 

0.07965 

0.08988 

2008 

2010 

0.10263 

0.1096 

0.09576 

2008 

2011 

0.16301 

0.16365 

0.14534 

2009 

2009 

0.07982 

0.07982 

0.07822 

2009 

2010 

0.11527 

0.11616 

0.114 

2009 

2011 

0.16658 

0.16481 

0.15079 

2010 

2010 

0.09616 

0.09616 

0.09363 

2010 

2011 

0.12217 

0.12119 

0.12048 

2011 

2011 

0.10249 

0.10249 

0.10633 


1. Test of Hypothesis 1 

The null hypothesis for this comparison is that the MAPE generated from 
experimental results is not smaller than the MAPE generated from a set of loss 
adjustment factors in which all values are 1.0. A paired two sample t-test is 
sufficient to test this hypothesis. The results of this t-test, displayed in Table 12, 
indicate that experimental results provide a better forecast than setting all loss 
adjustment factors equal to 1.0, at a 90% confidence level. 
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Table 12. t-Test: Paired Two Sample for Means, to test primary hypothesis 1: 

the MAPE generated from experimental results is not smaller than 
the MAPE generated from a set of loss adjustment factors in which 
all values are 1.0. 



Experimental 

Results 

hypothesis 1 

Mean 

0.100917333 

0.104296667 

Va riance 

0.001133595 

0.001639373 

Observations 

15 

15 

Pearson Correlation 

0.986185896 


Hypothesized Mean Difference 

0 


degrees of freedom 

14 


t Stat 

-1.426505455 


P(T<=t) one-tail 

0.087820762 


t Critical one-tail 

1.761310115 



2. Test of Hypothesis 2 

The null hypothesis for this comparison is that the MAPE generated from 
experimental results is not smaller than the MAPE generated from a set of loss 
adjustment factors in which values for projection year one are equal to 1.0, year 
two values are 1.041, year three values are 1.059, year four values are 1.085, 
and year five values are 1.114. Again, a paired two sample t-test is sufficient to 
test this hypothesis. The results of this t-test, displayed in Table 13, indicate that 
experimental results provide a better forecast than applying OPNAV N14’s loss 
adjustment factors describing a slowly improving economy (setting values for 
projection year one equal to 1.0, year two values are 1.041, year three values are 
1.059, year four values are 1.085, and year five values are 1.114.), at a 90% 
confidence level. 
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Table 13. 


t-Test: Paired Two Sample for Means, to test primary hypothesis 2: 
the MAPE generated from experimental results is not smaller than 
the MAPE generated from a set of loss adjustment factors describing 
a slowly improving economy. 



Experimental 

Results 

hypothesis 2 

Mean 

0.100917333 

0.10556 

Va riance 

0.001133595 

0.00173016 

Observations 

15 

15 

Pearson Correlation 

0.982063123 


Hypothesized Mean Difference 

0 


degrees of freedom 

14 


t Stat 

-1.691014156 


P(T<=t) one-tail 

0.056482551 


t Critical one-tail 

1.761310115 



The designs of experiments used in this thesis target analysis of data from 
base years 2007 through 2011, and this thesis has proven that changing loss 
adjustment factor values can generate a better forecast (measured by MAPE) 
than applying the same values to all loss adjustment factors. It is desirable to test 
these results against forecasts not included in the original analysis, but at this 
time additional data is unavailable. Nonetheless, this research demonstrates the 
benefits of applying data farming to OSAM, both to validate the model and 
improve it for future use. The fifth and final chapter of this thesis delineates 
significant findings and suggestions for future research. 
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V. CONCLUSIONS AND RECOMMENDATIONS 


This thesis demonstrates the potential to assess and improve OSAM with 
insights provided through data farming. This chapter summarizes specific 
findings related to SWO and HR loss adjustment factors, and suggests direction 
for further exploration of OSAM’s input parameters. OSAM utilizes input 
parameters contained in 60 database files; while this research varied a relatively 
small number of parameters in only one of those files, the methods employed to 
measure accuracy, generate a better forecast, and test specific research 
questions lay the foundation for countless further applications to OSAM and to 
other models owned by OPNAV N14. This thesis takes necessary first steps to 
explore the applicability of data farming, and the results suggest numerous 
applications for future research to undertake. 

A. SIGNIFICANT FINDINGS 

1. Assessment of Loss Rate Variation 

OSAM implements loss rate variation over a forecast period through the 
multiplication of historical loss rates by a loss adjustment factor. There is a 
potential to model unique loss adjustment factors for each combination of 
designator, projection year, and years of commissioned service (YCS). The 
rigorous numerical analysis in this thesis comparing simulation forecasts to 
historical inventories confirms that in many cases, officer loss rates next year will 
be similar to loss rates this year. The average experimentally determined SWO 
and HR loss adjustment factors for the first projection year range between 0.961 
and 1.031, and the 95% confidence interval includes the value 1.0 for every 
analyzed loss adjustment factor. This finding is in keeping with common practice 
in running OSAM simulations. 

Forecast analysis in projection years two through five yields less 
consistent results across the set of HR and SWO loss adjustment factors varied. 
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Recommended SWO Lieutenant (03) loss adjustment factors are less than 1.0 in 
every projection year, and multiple recommended values are very close to 1.0 
even in later projection years. For loss adjustment factors that deviate from 1.0, 
there seems to be a general upward trend with increasing projection years. 

2. Validation of Results 

This research delivers a suggested value to apply for each factor in future 
OSAM runs, and provides a Cl for these factors. The complex interactions 
between officer communities, ranks, and overall inventory make validation of 
individual loss adjustment factors impractical. A validation of the entire solution 
set is feasible, and demonstrates that experimentally determined loss adjustment 
factors yield a better forecast than holding parameters constant equal to 1.0, at a 
90% confidence level. This validation also affirms that experimental results yield 
a better forecast than varying all factors identically in each projection year 
according to OPNAV N14’s set of loss adjustment factors describing a slowly 
improving economy (1.0 in year one, 1.041 in year two, 1.059 in year three, 
1.085 in year four, and 1.114 in year five), again at a 90% confidence level. 

3. Loss Rate Variation for Declined Lateral Transfer Applicants 

As part of this research, a modification of OSAM adds a fictitious 
designator, 902x. This designator is adaptable to pursue various research 
questions, and for this thesis represents SWOs declined the opportunity to lateral 
transfer. Analysis of the first two experimental designs determines optimal values 
for 902x loss adjustment factors, and the final design of experiments executed 
twice, once with these experimentally determined values, and once with all 902x 
factors equal to their corresponding SWO factors. Analysis of this experiment 
reveals that there is no need to model loss adjustment factors differently for 
declined lateral transfer applicants. In fact, using the experimentally determined 
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902x values does not provide a more accurate forecast, as measured by MAPE, 
than applying the appropriate SWO values to these factors, at a 95% confidence 
level. 

This result at first seems counterintuitive, since earlier research (Kleyman 
& Parcell, 2010) shows that SWOs declined the opportunity to lateral transfer are 
more likely to leave the Navy than SWOs who never applied for lateral transfer. 
With this background in mind, it is important to recall that this hypothesis test 
indicates there is no benefit to varying loss adjustment factors distinctly for 902x 
officers, but does not consider the underlying loss rates built into the model. 
Notably, OSAM models 902x officers with unique historical loss rates, so this 
interesting finding is insufficient to suggest eliminating the 902x designator from 
the model. 

4. Applicability of Data Farming to OSAM 

One objective of this thesis is to explore the potential for continued 
application of data farming techniques to officer inventory projection. OSAM’s 
adaptation for use in a data farming environment enabled multiple simulations to 
run without operator interaction. This capability provides an opportunity to run 
countless variations of experimental designs in future explorations of the model. 
Data farming is unquestionably a useful technique for exploring OSAM. 

B. RECOMMENDATIONS FOR FUTURE RESEARCH 

1. Additional Base Years 

A noted limitation of this research is the lack of aligning forecast and 

historical data for early fiscal years and later projection years in the time period 

observed. The analytical approach employed adjusts for this information 

shortage by considering overlapping meta-models. The analysis of forecasts for 

additional base years is an ideal solution to this data scarcity. In addition to 

improving the accuracy of results, an expansion of this thesis to additional base 
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years provides an opportunity to observe forecast variation over additional 
economic and political environments, which may influence the retention behavior 
of some Navy officers. 

Analysis of forecast accuracy over a longer time period should reduce the 
number of meta-models needed to determine recommended loss adjustment 
factors. Additional data for each meta-model should also enable assessment of 
which set of meta-model groupings is most effective in determining 
recommended loss adjustment factors. An exploration of meta-models is 
particularly useful to OPNAV N14, as this methodology may apply to future 
versions of OSAM as a built-in accuracy check. 

2. Modeling Distinct Loss Rates for Declined Lateral Transfer 
Applicants 

This research concludes there is no need to model loss adjustment factors 
for declined lateral transfer applicants uniquely from the loss adjustment factors 
of officers who did not apply from lateral transfer. A logical next step is to assess 
whether there is value in modeling these declined lateral transfer applicants as a 
unique designator at all. As noted previously, OSAM models 902x officers with 
unique historical loss rates. The implementation of the 902x designator in OSAM 
is at the user’s discretion, and its definition is alterable according to the 
researcher’s needs. Running a DOE twice (once with 902x historical loss rates 
determined appropriately for SWOs declined the opportunity to lateral transfer, 
and once with 902x historical loss rates equal to corresponding SWO historical 
loss rates), will provide the data necessary to resolve this research question. 
Past research suggests these officers have unique promotion and loss 
probabilities. Application of data farming to this problem can determine whether 
these differences are significant enough to warrant modeling a unique 
designator. Application of this methodology to additional officer communities may 
determine which, if any, communities benefit from modeling this category of 
officers uniquely. 
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3. 


Additional Designators 


This research studies the loss adjustment factors for four of the 74 
designators modeled by OSAM. There is value in repeating this study for other 
designators. The methodology applied in this thesis is appropriate to recommend 
loss adjustment factors for the designators studied, but all officer communities 
interact in OSAM to meet the total inventory requirements imposed on the entire 
Navy Officer Corps. A similar study encompassing more designators may 
recommend different values for the loss adjustment factors studied in this thesis, 
due to such interactions. Data farming methods are ideal for tackling this type of 
problem expansion, enabling exploration of a complex, unwieldy design space. 

Data farming methods continue to evolve, and there are some potential 
limitations imposed by computing power and designs of experiments for large 
numbers of input parameters. OSAM includes 2,025 loss adjustment factors, and 
the variation of all factors simultaneously may present challenges in these areas. 
Exploration of the design space is possible without unique variation of all 2,025 
factors, with information about which designators have loss rates that vary 
similarly from year to year. Grouping designators together by categories of 
similar behavior could enable a study of all loss adjustment factors 
simultaneously. For instance, it may be appropriate to assign the same loss 
adjustment factors to all Limited Duty Officers (LDOs), a separate set of loss 
adjustment factors to all nurses and Medical Service Corps (MSC) officers, and 
yet another set of loss adjustment factors to all Restricted Line (RL) officers. A 
numerically determined solution set for all loss adjustment factors in OSAM could 
greatly improve the forecast accuracy of mid-term inventory projections. 

4. Model MAPE by Designator 

This thesis assesses the quality of forecasts by measuring mean absolute 

proportional error (MAPE), proportioned on rank. Application of this measure of 

effectiveness to more specific modeling may yield additional findings of interest 

to both OPNAV N14 and individual officer communities. For example, 

53 



proportioning MAPE on both designator and rank would enable the building of 
separate SWO and HR MAPE models. Both sets of models should still include 
loss adjustment factors from both communities, as their inventories interact 
during the simulation. The unique solution sets generated from this additional 
analysis may provide insight on the loss behavior of individual communities, and 
can identify the optimal set of loss adjustment factors for a goal of maximizing 
forecast accuracy of a single officer community. Similarly, both communities’ 
models could be considered together, weighting the MAPE models appropriately 
according to the Navy’s need to meet one community’s inventory needs more 
than another. This task requires significant input from Navy personnel subject 
matter experts, or consideration of many different weighting combinations. This 
approach to analysis has great potential for informing policy decisions, but the 
selection of appropriate weights for different officer communities is certain to vary 
widely between subject matter experts. 

5. Weighting Forecast Accuracy 

Accurate projection of Navy officer inventory is a goal of OSAM, and this 
thesis focuses on enhancing efforts to achieve this objective. In addition to 
overall forecast accuracy, there are many officer inventory goals that individual 
communities strive to meet. For instance, the SWO community needs a minimum 
number of Lieutenants at seven YCS to fill department head afloat billets; SWO 
accession plans intend to meet this need, even though this may result in more 
Ensigns than needed in the interim (Monroe & Cymrot, 2004). Data farming 
OSAM and analyzing MAPE calculations for very specific combinations of officer 
attributes can potentially improve projection to meet such specific goals. 

C. IMPACT OF RESEARCH 

OPNAV N14 gleans multiple benefits from this research, both in the short 
term and the long term. Analysts at N14 have envisioned building an automated 
validation or accuracy check capability into current and future forecasting 
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models, and will explore the use of MAPE as a forecast accuracy metric. The 
methodology utilized in this thesis seems a promising means of accomplishing 
this goal. 

This research highlights some weaknesses of OSAM that N14 can 
improve upon. One recommendation of this study is to include more historical 
data to adequately measure accuracy; N14 plans to make this a requirement in 
future model development. This thesis also identifies OSAM’s overestimation of 
SWO Lieutenant loss rates in all projection years. The loss estimation process is 
a key step in personnel forecasting, and N14 strives to model losses well. The 
next update of OSAM will strive to correct this weakness, either including more 
explanatory historic information (e.g., years of total service), or using new rate 
generation techniques (e.g., machine learning, Bayesian, or agent-based 
behavioral models). 

To leverage the conclusion that forecast accuracy can improve with 
experimentally determined loss adjustment factors, N14 will ensure that future 
models retain the capability for users to modify loss rates. This potential benefit 
lends itself to future research, particularly the investigation of loss rates for 
additional specific officer communities. 

This thesis, the first application of data farming concepts to OSAM, lays 
the groundwork for continuing efforts in this area. The conclusions of this 
research provide actionable steps for OPNAV N14 to pursue. Further, the 
implementation of designs of experiments, selection of a forecast accuracy 
metric, and development of a multi-faceted analytical approach employed in this 
study serve as a roadmap for future exploration of OSAM. 


55 



THIS PAGE INTENTIONALLY LEFT BLANK 


56 



APPENDIX 


This appendix describes the data farming components used to data farm 
OSAM. The information included here is derived from the notes of Stephen 
Upton, who developed the code used and executed all experiments in this thesis 
on the computer clusters of the SEED Center for Data Farming. 

The current implementation of data farming OSAM is a limited version, in 
that the development of code and other artifacts is specifically to assist in 
completion of this thesis. The hard coding of some settings supports timely 
implementation of OSAM in a data farming environment. In particular, the design 
points used in the experiments varied only factors in the loss_adj.dbf file, based 
on settings for fiscal year, grade, and designator. Using the current 
implementation, one can vary a factor for any combination of fiscal year, grade, 
or designator contained in the loss_adj.dbf file. Extension of data farming to 
varying factors in any other input file will require editing the implementation code. 

OSAM simulations run via a Microsoft Visual Fox Pro Version 9 (VFP9) 
executable file. An additional VFP9 file, dfosam.exe, assists in data farming 
OSAM. A licensed copy of VFP9 is necessary to edit the OSAM input files, as 
indicated by designs of experiments. In addition to this new VFP9 executable file, 
several other elements are part of the vital infrastructure for data farming OSAM. 
Table 14 provides a summary of these components. 

Each OSAM run saves the output files, Multi-Year Summary.dbf and 
flow_pt.dbf, in a directory named for the appropriate design point. These are the 
files containing data to be processed and analyzes as described in Chapter III. 
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Table 14. 


Summary of components utilized in the implementation of OSAM in a 
data farming environment. 


Component 

Description 

stage.OSAM.run.R 

this code creates the Output and submit directories, then 
creates a DP_X directory for all DPs in the design, where 
X is the design point or row number of the DOE file. It 
then creates condor_submit files for all the DPs and puts 
them in the submit directory, (this code needs 4 
arguments: [1] study.dir is the directory location, [2] 
basecase.name is the zipped contents of an OSAM 
scenario, including the OSAM executable file, [3] 
start.year, [4] doe.file.name) 

submit-template.data 

this is the condor submit template, with BASECASE 
STARTYEAR DOEFILE DPNUM and STUDY variables that 

get replaced with appropriate values by 
stage.OSAM.run.R code 

dfOSAM.R 

Generates the appropriate factors.csv file used by 
dfosam.exe using the factor.map.csv file, the baseyear, 
startyear, and DP number as input 

dfosam.exe 

short VFP9 executable that updates the loss_adj.dbf file 
based on the factors.csv file 

dfOSAM.bat 

calls dfOSAM.R, dfosam.exe and the OSAM executable in 
proper order of execution on the compute node 
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